Data management in R
2 Jun 2025
Disclaimer
I owe a debt of gratitude to many people as the thoughts and code in these slides are the process of years-long development cycles and discussions with my team, friends, colleagues and peers. When someone has contributed to the content of the slides, I have credited their authorship.
These materials are generated by Gerko Vink, who holds the copyright. The intellectual property belongs to Utrecht University. Images are either directly linked, or generated with StableDiffusion or DALL-E. That said, there is no information in this presentation that exceeds legal use of copyright materials in academic settings, or that should not be part of the public domain.
Warning
You may use any and all content in this presentation - including my name - and submit it as input to generative AI tools, with the following exception:
- You must ensure that the content is not used for further training of the model
Slide materials and source code
Materials
- course page: www.gerkovink.com/sur
- source: github.com/gerkovink/sur
Recap
Yesterday we have learned:
- How to use
RandRStudio - How to install packages
- How to use simple data containers
- How to do subsetting in base
Rwith[ ] - How to use logical operators to subset data
Today
Today we will learn how to:
This lecture
- The blueprint of
R - Code conventions en style
- Basic data manipulation
- Basic analysis (correlation & t-test)
- R model formula and classes
- Pipes
New packages we use
New functions
transform(): changing and adding columnsdplyr::filter(): row-wise selection (of cases)dplyr::arrange(): order rows by values of a column or columns (low to high)table(): frequency tablesclass(): object classlevels(): levels of a factororder(): data entries in increasing orderhaven::read_sav(): import SPSS datacor(): bivariate correlationsample(): drawing a samplet.test(): t-test
The blueprint of R
Layers in R
There are several ‘layers’ in R. Some layers you are allowed to fiddle around in, some are forbidden. In general there is the following distinction:
- The global environment.
- User environments
- Functions
- Packages
- Namespaces
Environments
The global environment can be seen as an olympic-size swimming pool. Everything you do has its place there.
If you’d like, you may create another, separate environment to work in.
- A user environment would by default not have access to other environments
Functions
If you create a function, it is positioned in the global environment.
Everything that happens in a function, stays in a function. Unless you specifically tell the function to share the information with the global environment.
See functions as a shampoo bottle in a swimming pool to which you add some water. If you’d like to see the color of the mixture, you’d have to squeeze the bottle for it to come out.
Packages
Packages have their own space.
- Everything needed to run the functions in a package is needly contained within its own space
- See packages as separate (mini) pools that are connected to the main pool (the global environment)
Loading packages
There are two ways to load a package in R
and
require() will produce a warning when a package is not found. In other words, it will not stop as function library() does.
Installing packages
The easiest way to install e.g. package mice is to use
Alternatively, you can also do it in RStudio through
Tools --> Install Packages
Namespaces
- Namespaces. These are the deeper layers that feed new water to the surface of the mini pools.
- Packages can have namespaces.
- Functions within packages are executed within the package or namespace and have access to the global environment.
- Objects in the global environment that match objects in the function’s namespace are ignored when running functions from packages!
R in depth
Workspaces and why you should sometimes save them
A workspace contains all changes you made to environments, functions and namespaces.
A saved workspace contains everything at the time of the state wherein it was saved.
You do not need to run all the previous code again if you would like to continue working at a later time.
- You can save the workspace and continue exactly where you left.
Workspaces are compressed and require relatively little memory when stored. The compression is very efficient and beats reloading large datasets from raw text.
History and why it is useful
R by default saves (part of) the code history and RStudio expands this functionality greatly.
Most often it may be useful to look back at the code history for various reasons.
There are multiple ways to access the code history.
- Use arrow up in the console. This allows you to go back in time, one codeline by one. Extremely useful to go back to previous lines for minor alterations to the code.
- Use the history tab in the environment pane. The complete project history can be found here and the history can be searched. This is particularly convenient when you know what code you are looking for.
Working in projects in RStudio
- Every project has its own history
- Every research project has its own project
- Every project can have its own folder, which also serves as a research archive
- Every project can have its own version control system
- R-studio projects can relate to Git (or other online) repositories
Modeling
Modeling in R
To model objects based on other objects, we use ~ (tilde)
For example, to model body mass index (BMI) on weight, we would type
The ~ is used to separate the left- and right-hand sides in a model formula.
For functions (or models), within models we use I() - For example, to model body mass index (BMI) on its deterministic function of weight and height, we would type
Example
Remember the boys data from package mice:
Example continued
Remember the boys data from package mice:
Why we really need I()
More efficient
It is ‘nicer’ to store the output from the function in an object. The convention for regression models is an object called fit.
The object fit contains a lot more than just the regression weights. To inspect what is inside you can use
Inspecting what is inside fit
Another approach to inspecting the contents of fit is the function attributes()
$names
[1] "coefficients" "residuals" "effects" "rank"
[5] "fitted.values" "assign" "qr" "df.residual"
[9] "na.action" "xlevels" "call" "terms"
[13] "model"
$class
[1] "lm"The benefit of using attributes() is that it directly tells you the class of the object.
Classes in R
Classes are used for an object-oriented style of programming. This means that you can write a specific function that - has fixed requirements with respect to the input. - presents output or graphs in a predefined manner.
When a generic function fun is applied to an object with class attribute c("first", "second"), the system searches for a function called fun.first and, if it finds it, applies it to the object.
If no such function is found, a function called fun.second is tried. If no class name produces a suitable function, the function fun.default is used (if it exists). If there is no class attribute, the implicit class is tried, then the default method.
Classes example: plotting without class
Classes example: plotting with class
Classes example: plotting with class
Classes example: plotting with class
Classes example: plotting with class
Classes example: plotting with class
Classes example: plotting with class
Why is plot different for class "lm"?
The function plot() is called, but not used. Instead, because the linear model has class "lm", R searches for the function plot.lm().
If function plot.lm() would not exist, R tries to apply function plot() (which would have failed in this case because plot requires x and y as input)
plot.lm() is created by John Maindonald and Martin Maechler. They thought it would be useful to have a standard plotting environment for objects with class "lm".
Since the elements that class "lm" returns are known, creating a generic function class is straightforward.
R-coding
The Google style guide
Naming conventions
File Names
File names should end in .R and, of course, be meaningful.
GOOD:
BAD:
Identifiers
Don’t use underscores ( _ ) or hyphens ( - ) in identifiers. Identifiers should be named according to the following conventions.
- The preferred form for variable names is all lower case letters and words separated with dots (variable.name), but variableName is also accepted;
- function names have initial capital letters and no dots (FunctionName);
- constants are named like functions but with an initial k.
Identifiers (continued)
variable.name is preferred, variableName is accepted
GOOD:avg.clicks
OK:avgClicks
BAD:avg_ClicksFunctionName
GOOD:CalculateAvgClicks
BAD:calculate_avg_clicks,calculateAvgClicks
kConstantName
Syntax
Line Length
The maximum line length is 80 characters.
# This is to demonstrate that at about eighty characters you would move off of the page
# Also, if you have a very wide function
fit <- lm(age ~ bmi + hgt + wgt + hc + gen + phb + tv + reg + bmi * hgt + wgt * hgt + wgt * hgt * bmi, data = boys)
# it would be nice to pose it as
fit <- lm(age ~ bmi + hgt + wgt + hc + gen + phb + tv + reg + bmi * hgt
+ bmi * wgt + wgt * hgt + wgt * hgt * bmi, data = boys)
#or
fit <- lm(age ~ bmi + hgt + wgt + hc + gen + phb + tv + reg
+ bmi * hgt
+ bmi * wgt
+ wgt * hgt
+ wgt * hgt * bmi,
data = boys)Indentation
When indenting your code, use two spaces. RStudio does this for you!
Never use tabs or mix tabs and spaces.
Exception: When a line break occurs inside parentheses, align the wrapped line with the first character inside the parenthesis.
Spacing
Place spaces around all binary operators (=, +, -, <-, etc.).
Exception: Spaces around =’s are optional when passing parameters in a function call.
or
Spacing (continued)
Do not place a space before a comma, but always place one after a comma.
GOOD:
Spacing (continued)
BAD:
# Needs spaces around '<'
tab.prior <- table(df[df$days.from.opt<0, "campaign.id"])
# Needs a space after the comma
tab.prior <- table(df[df$days.from.opt < 0,"campaign.id"])
# Needs a space before <-
tab.prior<- table(df[df$days.from.opt < 0, "campaign.id"])
# Needs spaces around <-
tab.prior<-table(df[df$days.from.opt < 0, "campaign.id"])
# Needs a space after the comma
total <- sum(x[,1])
# Needs a space after the comma, not before
total <- sum(x[ ,1]) Spacing (continued)
Place a space before left parenthesis, except in a function call.
GOOD:
BAD:
Extra spacing
Extra spacing (i.e., more than one space in a row) is okay if it improves alignment of equals signs or arrows (<-).
Do not place spaces around code in parentheses or square brackets.
Exception: Always place a space after a comma.
Extra spacing
GOOD:
BAD:
In general…
Use common sense and BE CONSISTENT.
The point of having style guidelines is to have a common vocabulary of coding
- so people can concentrate on what you are saying, rather than on how you are saying it.
If code you add to a file looks drastically different from the existing code around it, the discontinuity will throw readers out of their rhythm when they go to read it. Try to avoid this.
Basic data manipulation
The cats data
Description (see ?cats): The heart weights (in g) and body weights (in kg) of 144 adult cats (male and female) used for digitalis experiments. Digitalis is a medication used to treat heart failure and certain types of irregular heartbeat.
The cats data
How to get only Female cats?
How to get only heavy cats?
Select cats with specified range of body weights
Working with factors
Working with factors
Changing the level names:
Sorting with dplyr::arrange()
dplyr::arrange()sorts the rows of a data frame according to the order of one or more of the columns.
Sorting with dplyr::arrange()
dplyr::arrange()sorts the rows of a data frame according to the order of one or more of the columns.
Combining matrices or dataframes
# add variable name for ID variable and change names of other variables
D <- dplyr::bind_cols(ID, cats)
names(D) <- c("ID", "Sex", "Bodyweigt", "Hartweight")
head(D) ID Sex Bodyweigt Hartweight
1 1 Female 2.0 7.0
2 2 Female 2.0 7.4
3 3 Female 2.0 9.5
4 4 Female 2.1 7.2
5 5 Female 2.1 7.3
6 6 Female 2.1 7.6Basic analysis
Correlation
With [, -1] we exclude the first column (female/male)
Correlation
Pearson's product-moment correlation
data: cats$Bwt and cats$Hwt
t = 16.119, df = 142, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
0.7375682 0.8552122
sample estimates:
cor
0.8041274 What do we conclude?
Correlation
T-test
Test the null hypothesis that the difference in mean heart weight between male and female cats is 0
Welch Two Sample t-test
data: Hwt by Sex
t = -6.5179, df = 140.61, p-value = 1.186e-09
alternative hypothesis: true difference in means between group Female and group Male is not equal to 0
95 percent confidence interval:
-2.763753 -1.477352
sample estimates:
mean in group Female mean in group Male
9.202128 11.322680 T-test
Linear regression
Remember the boys data from package mice.
If we want to regress BMI (bmi) on weight (wgt), we would use the lm() function and the model formula bmi ~ wgt.
Scatterplot
Add a quadratic term
Why we need I()
Inside an R model formula, the +, :, and ^ operators are text objects and therefore behave differently than when used in calculations with numeric vectors. Using I() converts these operators to their numerical meaning.
Output of the lm() class
It is ‘nicer’ to store the output from the function in an object. The convention for regression models is an object called fit.
The object fit contains a lot more than just the regression weights. To inspect what is inside you can use
Inspecting what is inside fit
Another approach to inspecting the contents of fit is the function attributes()
$names
[1] "coefficients" "residuals" "effects" "rank"
[5] "fitted.values" "assign" "qr" "df.residual"
[9] "na.action" "xlevels" "call" "terms"
[13] "model"
$class
[1] "lm"The benefit of using attributes() is that it directly tells you the class of the object.
Classes in R
Classes are used for an object-oriented style of programming. This means that you can write a specific function that
- has fixed requirements with respect to the input.
- presents output or graphs in a predefined manner.
When a generic function fun is applied to an object with class attribute c("first", "second"), the system searches for a function called fun.first and, if it finds it, applies it to the object.
If no such function is found, a function called fun.second is tried. If no class name produces a suitable function, the function fun.default is used (if it exists). If there is no class attribute, the implicit class is tried, then the default method.
Classes example: plotting without class
Classes example: plotting with class
Classes example: plotting with class
Classes example: plotting with class
Why is plot different for class "lm"?
The function plot() is called, but not used. Instead, because the linear model has class "lm", R searches for the function plot.lm().
If function plot.lm() would not exist, R tries to apply function plot() (which would have failed in this case because plot requires x and y as input)
plot.lm() is created by John Maindonald and Martin Maechler. They thought it would be useful to have a standard plotting environment for objects with class "lm".
Since the elements that class "lm" returns are known, creating a generic function class is straightforward.
Pipes
This is a pipe:
It effectively replaces head(read_sav("boys.sav")).
Why are pipes useful?
Let’s assume that we want to load data, change a variable, filter cases and select columns. Without a pipe, this would look like
With the pipe:
Benefit: a single object in memory, the steps are easy to follow and understand.
With pipes
Your code becomes more readable:
- data operations are structured from left-to-right and not from in-to-out
- nested function calls are avoided
- local variables and copied objects are avoided
- easy to add steps in the sequence
Keyboard shortcut:
- Windows/Linux:
ctrl + shift + m - Mac:
cmd + shift + m
What do pipes do:
What do pipes do:
Pipes create functions without nesting:
f(x) becomes x %>% f()
Pipes create functions without nesting
f(x, y) becomes x %>% f(y)
hgt wgt bmi
hgt 1.0000000 0.6100784 0.1758781
wgt 0.6100784 1.0000000 0.8841304
bmi 0.1758781 0.8841304 1.0000000 hgt wgt bmi
hgt 1.0000000 0.6100784 0.1758781
wgt 0.6100784 1.0000000 0.8841304
bmi 0.1758781 0.8841304 1.0000000Pipes create functions without nesting
h(g(f(x))) becomes x %>% f %>% g %>% h
Useful: outlier filtering
[1] 144# select the observations that fall outside the +/- SD around the mean
cats.outl <-
cats %>%
dplyr::filter(Hwt > mean(Hwt) + 3 * sd(Hwt) | Hwt < mean(Hwt) - 3 * sd(Hwt))
nrow(cats.outl)[1] 1# select the observations that fall within +/- 3 SD around the mean
cats.without.outl <-
cats %>%
dplyr::filter(Hwt < mean(Hwt) + 3 * sd(Hwt) & Hwt > mean(Hwt) - 3 * sd(Hwt))
nrow(cats.without.outl)[1] 143More pipe stuff
Pipes and the R model formula ~
Sometimes the data we want to use, are “piped” in the wrong argument, see e.g.:
This leads to an error:
Error in as.data.frame.default(data) :
cannot coerce class ‘“formula”’ to a data.frame
The %>% pipe operator is intended to work with functions where each result is forwarded on to the first argument of the next function. For the function lm() the first argument is the model formula, which is a text object and not the data frame.
Solution 1
We can use the . symbol to act as placeholder for the data:
Solution 2: the exposition %$% pipe
Solution 2: the exposition %$% pipe
The exposition %$% pipe exposes the content(s) of the data frame (the variables) to the next function in the pipeline.
Debugging pipelines
Sample 3 positions from the alphabet and show the position and the letter. If you don’t know what’s going on, run each statement separately!
Performing a t-test in a pipe
Welch Two Sample t-test
data: Hwt by Sex
t = -6.5179, df = 140.61, p-value = 1.186e-09
alternative hypothesis: true difference in means between group Female and group Male is not equal to 0
95 percent confidence interval:
-2.763753 -1.477352
sample estimates:
mean in group Female mean in group Male
9.202128 11.322680 is the same as
Storing a t-test from a pipe
Welch Two Sample t-test
data: Bwt by Sex
t = -8.7095, df = 136.84, p-value = 8.831e-15
alternative hypothesis: true difference in means between group Female and group Male is not equal to 0
95 percent confidence interval:
-0.6631268 -0.4177242
sample estimates:
mean in group Female mean in group Male
2.359574 2.900000 
Gerko Vink @ Anton de Kom Universiteit, Paramaribo